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AN INTRODUCTION 


ver the past six years, the Pullias Center for Higher Education has been engaged in research 
() designed to better understand the potential of digital tools to increase access to college. Our most 

recent $3.2 million First in the World (FITW) federal grant has enabled our Center to implement 
and evaluate an online game-based intervention in schools across the state of California. Beyond 
examining the effects of the game tool on college outcomes, we also collected and analyzed robust mixed 
methods data pertaining to college-going culture and digital infrastructure in California high schools 
serving low-income youth. 


During this final year of the project and in preparation for the dissemination of findings, we have partnered 
with the California Student Aid Commission (CSAC) and the Campaign for College Opportunity to host 

a convening on April 6, 2018. The event will feature speakers from the academic, policy, and advocacy 
sectors with extensive knowledge of the educational data gaps that currently prevent researchers, school 
practitioners, state agencies, and policymakers from meeting the needs of students, particularly low- 
income and first-generation college students, from across the state. These data gaps are most visible at the 
intersection of digital equity, college access, and college completion. Our goal for this meeting is to provide 
the answers to three questions in order to develop a better understanding of California's educational data 
landscape and how it can be altered to better meet the needs of students: 


1. Which data do we lack (to inform the cultivation of postsecondary education and career pathways)? 
Why are these data lacking (what legal, regulatory, logistical barriers prevent data from being 
collected, recorded, or shared)? 

3. Ifattained, how could we use the data to improve educational outcomes? 


The purpose of this document is to give a brief background on the Pullias Center’s FITW study, introduce 
the data challenges we encountered during the study, and present four possible causes of these challenges 
based on interview and survey data. 


DATA ACCESS, SHARING & TRANSPARENCY 


three college outcomes: (1) Free Application for Federal Student Aid (FAFSA) and California Dream 

Act (CADA) Application completion, (2) college applications, and (3) college enrollment. For these 
purposes, we collected a wide range of data, including demographic, GPA, A-G units completed, FAFSA 
and CADA completion (through CSAC), and college application data. Throughout the study, we 
encountered several challenges regarding data access, sharing, and transparency which are suggestive of 
larger state-wide problems impacting students, K-12 schools and districts, and higher education 
institutions. 


Te FITW study team initially planned to evaluate the impact of our game-based intervention on 


Though addressing data gaps was not an intended focus of the study, the topic was nonetheless 
unavoidable; in addition to our own difficulties obtaining necessary data for the study outcomes, in our 
qualitative research with school faculty and staff, we learned that schools themselves struggled to collect 
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and obtain data that could be used to improve curricula, teaching, student performance, and even college 
outcomes. For these reasons, we analyzed data obtained through interviews and survey results in order to 
identify root causes. We have organized findings into four categories of possible causes: collection of data, 
data constraints, communication, and adaptation. We briefly explore each category below along with 
examples from study data. 


COLLECTION OF DATA 


he data collection category includes not only which data are collected from students, but also in 
what form the data are collected, and general data collection practices within and across schools and 
school districts. 


Data collected at the school level in California varied widely by school and district. During the 2016- 

2017 school year, we issued a survey to participating schools about their college readiness data collection 
practices. Among the questions asked of site contacts - typically college or career counselors — was whether 
they collected the following information: where students apply, are accepted, plan to go, their summer 
plans after graduation, and whether they enroll in college in the fall. We also inquired about how the data 
were collected: were they self-reported (hard or e-copy)? Obtained through individual meetings? Through 
data tracking software such as Naviance? We found that very few schools or districts across California had 
systematic data collection procedures in place. Many of the schools represented in the survey collected 
data informally, meaning that data were collected for some students but not others. Given the frequency of 
self-reported student data and inconsistency in data collection practices (e.g. one-on-one interviews versus 
surveys), a concern of the team across the twenty participating school districts was whether the college 
readiness data collected in schools were reliable. 


In interviews, we asked site contacts directly about their experiences with college-readiness data collection 
and tracking. Participants described discrepancies within their schools in how data were recorded (e.g., 
some staff used Excel spreadsheets while others collected handwritten notes) and stored (we heard stories of 
misplaced handwritten notes and discoveries of years-old data from long-gone teachers). One site contact 
replied that he did not have access to National Student Clearinghouse data, and he did not attempt to 

track students after graduation: “I don't track, I just don’t. I am not good at keeping in touch with people 

in general and I don't really use Facebook and all that kind of stuff” Another counselor disclosed that her 
school did not collect college readiness data: 


college readiness—we dont track that, other than just looking at grades and test scores. College 
going—that’s been our struggle. We can get them at the end to list, ‘this is where they’re going, but 
were finding that between graduating in June and August, they don’t all end up there. 


Though most schools in the study collected some form of college readiness data on their students, these 
data were often self-reported and the practices used to collect them were seldom consistent among school 
staff. Because few schools maintained structured and systematic data collection procedures, even where 
data are being collected, they may be unreliable or have little value to schools in the long run. 
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DATA CONSTRAINTS 


guidance. Schools that are understaffed or experience high staff turnover are likely to struggle to 

track students for the purpose of school improvement and student outcomes. Within this category, 
we found vast structural differences across California schools: from a lack of summer assistance for seniors 
(despite rampant and established problems with “summer melt”) to high student to college counselor ratios 
or no designated college counselors at all. For example, one staff member explained that she was limited in 
her ability to assist seniors applying to college because her position was only ten months long: 


D ata constraints are connected to limitations of job profiles and/or resources dedicated to college 


my last day of work is the students’ last day of school, so if there was a student that might have 
had...something go wrong in that process, maybe was confused about when to sign up for 
orientation or anything like that, then...then when I’m not here, they might fall through the cracks. 


Another staff member discussed the resource limitations of her school, stating that, even though the 
academic counselors meet with each senior to develop a college plan, she did not think the school ever 
had a college counselor, per se: “we were our own college counselors for each student.” Still another staff 
member expressed uncertainty about what to do with the data once they were collected; that participant 
explained that she does not maintain files on students’ college readiness because there was no one in the 
school to whom she should report: “I know where [the students are] at, or where they’re going, or where 
they applied,” but “I haven't kept files...because we're not—we don't have [anyone] to report to”” These and 
other examples of staff constraints reflect that there is likely no perfect solution for every school, but that 
all schools may benefit from certain policies across the board, including clear and stringent data collection 
procedures. 


COMMUNICATION 


hallenges to effective communication came through in many interviews with site contacts, usually 

revolving around post-graduation data. The most common problem expressed by interview 

participants was that school staff struggled to obtain data on their students once they graduated or 
left the school, which made tracking college outcomes difficult. The problems typically involved barriers 
to data sharing relating to the Family Educational Rights and Privacy Act (FERPA) of 1974. Designed to 
protect the privacy rights of students, FERPA prevents educational institutions receiving federal dollars 
from sharing their students’ data absent written consent from those students. In practice, FERPA effectively 
bars K-12 schools from obtaining data from former students, even where those schools might fit into a 
FERPA exception (i.e., school officials with a legitimate educational interest). 


In California, where students frequently go on to attend University of California (UC), California State 
University (CSU), or community college schools, K-12 counselors often encounter challenges obtaining 
enrollment data on their former students. One participant stated: “last year, I got [information on the 
students] that [UC and CSU] had accepted, but they won't tell you who is going to attend. Now, how am 
I supposed to track where they're going if I can't even get that information from them?” Another school 
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counselor confirmed that she cannot access UC or CSU data because of FERPA blocks: “once they start 
applying to college I only can track them anecdotally because I don't have access to their accounts.” Another 
echoed this concern: “we track where they say they’re going, but not what happens afterwards;” and 
another: “we can't keep up with [students] unless it’s voluntary, unless they volunteer to [offer information] 
and they have to come back for us to even be able to contact them.” 


These communication issues may be problematic from a bigger-picture perspective in that they prevent 
meaningful partnerships between K-12 schools and higher education institutions. Where state universities 
are unable to share data about student enrollment or performance, schools and districts are unable to use 
that data to improve their academic and college curricula. 


ADAPTATION 


he adaptation category refers to school staff’s ability to adapt to technological, curricular, or 
personnel changes. Proficiency with new technology, as well as problems associated with staff 
training and high turnover, had an impact on a school’s ability to collect, manage, and utilize data. 


Most often these problems manifested as technological struggles; we encountered examples of schools 
adopting new software and systems designed to facilitate student data collection, such as Naviance, Google 
Sheets, or even shifts from hard-copy to online surveys. But these changes were not always successful, 

and in schools where they were successful, not all staff were equally on board or comfortable using the 
technology. For example, in schools and districts which have access to costly Naviance subscriptions, we 
found that staff were not always trained to use them. At one school with access to Naviance, the site contact 
revealed that “[Naviance] is underutilized. We dont do a lot on there...the technical difficulties make it too 
challenging. We just do our hard copies.” At a school which was new to Naviance, a site contact explained 
that the system was a challenge and that so far it was “a little intimidating” As technology evolves, schools 
and districts adopt new systems for data collection and tracking, but it is unclear who trains staff to use new 
technologies and how often. 


CONCLUSION 


he purpose of this document is to summarize observations from a three-year long intervention 

across schools in California to learn more about the impact of digital tools on college access. Based 

on challenges we encountered throughout implementing the intervention, we found that schools 
struggled to collect and utilize data effectively in the college-readiness realm with potentially far-reaching 
policy implications. We also documented instances, such as CSAC’s efforts to share FAFSA and CADA 
completion data with schools, where data practices supported positive college-going trajectories. Based on 
our observations, we believe that college-readiness data can be used to improve college-going outcomes 
for students. However, without participation and buy-in from a wide range of stakeholders —- K-12 schools 
and districts, postsecondary institutions, and government agencies — and without systemic measures for 
ensuring data security and accountability for reporting, schools run the risk of not adequately supporting 
students in their pursuit of postsecondary education. 


For more information about the project visit: https://pullias.usc.edu/digitalequity/ 
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